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METHOD AND APPARATUS FOR 
COLORMETRIC CHANNEL BALANCING FOR SOLID STATE IMAGE SENSOR 
USING TIME DIVISION MULTIPLEXED SAMPLING WAVEFORMS 
Field of the Invention 

5 This invention is related to the field of use of image sensor operation. More 

particularly, this invention is directed to a method and apparatus for colormetric channel 
balancing for solid state image sensor using time division multiplexed sampling 
waveforms. 
Background 

10 Video image capture using a digital camera has become very prevalent. Video 

capture may be used for such applications as video conferencing, video editing, and 
distributed video training. Many components of hardware and software need to work 
seamlessly to both get the video data (also referred to as a video "stream") from the 
hardware through the various layers of software on the personal computer (PC) and made 

1 5 visible on a display medium such as a monitor. 

Producing high quality images in color is desirable for electronic still image 
capture and video systems. Conventional methods of sensing and producing an image in 
color include the use of electronic charged-coupled device (CCD) or complimentary 
metal oxide semiconductor (CMOS) sensor arrays exposed to incident light. The sensor 

20 array is an array of photosites or pixels, each photosite loosely defined as a region 

containing photodetecting circuitry that includes, for example, photodiodes or photogates 
and associated processing circuitry. In some cases, a color filter array (CFA) is pasted or 
otherwise processed over the array of photodetecting circuits, such that each 
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photodetecting circuit is covered by a bandpass optical filter that lets light of a particular 
color pass into and thus be detected by the corresponding circuit. Sensor arrays typically 
have two or more colors distributed evenly, or according to other schemes, in the array of 
photosites,' where a group of photosites may be assigned a particular color. 

To obtain sharp color images from such sensor arrays over a broad range of 
illuminants, it may be desirable to control the amount of light energy that is incident on 
the sensor array. One technique for doing so is to modulate the incident light using a 
physical shutter having variable timing, as in a conventional film camera. The idea of a 
shutter has been applied to digital solid state cameras in the form of an electronic shutter. 
For example, in CMOS sensor arrays, the electronic shutter is typically a transistor that 
couples a photo-detecting element, such as a photodiode, to a charge storage element, 
such as a capacitor, in each photosite. The electronic shutter transistor operates as a 
switch in response to a shutter control signal that specifies the "exposure time" by 
defining the time interval during which the shutter transistor is turned on allowing charge 
to transfer from the exposed photodiode and accumulate in the capacitor. Alternatively, 
the shutter transistor may be used to drain pre-stored charge from the capacitor. 

Typically, prior art systems that use such electronic shutters provide a single 
shutter control signal for the entire sensor array. Thus, in most prior art systems, 
photosites of different colors receive the same exposure time. Although some systems 
permit different exposure times for each color, they do so by requiring separate and 
sequential exposures for each color, typically by using a mechanical apparatus to change 
the color filter over a panchromatic sensor array. Taking sequential exposures for each 
color effectively precludes taking color pictures of scenes that include motion. 
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Moreover, such a complicated mechanical apparatus, normally used in devices such as 
flatbed scanners, would present reliability problems and would not be easily adaptable to 
portable image capture systems such as a digital handheld camera. 

Although the technique of using a single shutter control line for the entire sensor 
5 array is relatively simple to implement, such an application may also yield poor quality 
images for illuminants other than broad daylight. Under low or uneven light conditions, 
colors for which there is a dearth of illumination are less apparent in the resulting image 
due to increased noise levels in the analog signals received from the photosites tuned for 
those "weaker" colors. Image processing algorithms may be used to somewhat improve 
f 10 image quality by canceling out the increased noise in the signals for the weaker colors. 

; : But such a software solution may introduce undesirable delays before the final image can 

'4 

^ be viewed. Moreover, the results of such software corrections are not always predictable 

, 3 

*\ or consistent. 

& Therefore, a hardware approach may be desirable to yield more consistent and 

y 15 predictable noise levels. Also, it is desirable to realize a color image system that can 
3 somewhat compensate for manufacturing variations in the myriad of photosites in a 

sensor array, including variations in the photodetecting circuitry and the bandpass filters 

of the CFA. 
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SUMMARY 

In one embodiment, a method including determining a first integration time for a 
first color channel and determining a second integration time for a second color channel, 
where the second integration time is shorter than the first integration time. Then, the 
second integration time is divided into a set of integration times. A first sensor is 
integrated over the first integration time; and, a second sensor is integrated over the set of 
integration times. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
The present invention is illustrated by way of example, and not by way of 
limitation, in the figures of the accompanying drawings and in which like reference 
numerals refer to similar elements and in which: 
5 Figure la illustrates an image sensor circuit according to a first embodiment of 

the present invention; 

Figure lb shows exemplary shutter control signals for each color, according to 
one embodiment of the present invention; 

Figure 2 shows an image sensor circuit according to a second embodiment of the 
1 0 present invention; 

Figure 3 shows an image capture system according to one embodiment of the 
present invention; 

Figure 4 illustrates an integration control unit of the image capture system of 
Figure 3 configured in accordance with one embodiment of the present invention. 
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DETAILED DESCRIPTION 
The present invention provides a method and apparatus for colormetric channel 
balancing for solid state image sensor using time division multiplexed sampling 
waveforms. In one mode of operation in performing an image capture, a set of 
5 integration (e.g., sample) signals is used, with a separate integration signal being used for 
each color. Each separate integration signal is time division multiplexed to allow all 
separate integration signals to provide sequential color sampling while minimizing total 
integration time delay disparity. By minimizing total integration time delay disparity, 
motion artifacts are also reduced. 

10 Figure la illustrates a block diagram of an image sensor circuit 100 according to 

one embodiment of the invention. The circuit 100 has a sensor array of pixels or 
photosites 111,112,..., arranged in rows and columns for that particular embodiment. A 
CFA (not shown) may be pasted or otherwise processed over the photo-detecting 
circuitry in the sensor array such that each photosite is in effect configured to detect 

15 incident light of a particular color, as indicated by the color label adjacent each photosite. 

An exemplary photosite 111 may have a green filter covering, and may include a 
row select input, an electronic shutter input, and an analog output. Not shown in each 
photosite is conventional read out circuitry that couples to a photo-detecting element such 
as a photodiode and allows the formation of a light-generated signal that is provided at 

20 the analog output. A single shutter control line is coupled to the shutter input of all 
photosites having the same color in a given row, where each control line receives an 
independent pulse-type shutter control signal, as shown in Figure lb. 
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The embodiment of the invention as circuit 100 also features a column Analog-to- 
Digital (AID) converter where the light-generated signals from a given column are 
multiplexed and converted by the column A/D converter into digital format one row at a 
time. Another scheme may use A/D converters for each row, so that the digital signals 
5 are read per column. In either case, the digital signals may then be subsequently stored 
and further processed. 

The array of circuit 100 employs one particular distribution of colors. However, 
the CFA can take on different embodiments with the colors being distributed across the 
sensor circuit according to other patterns. For example, Figure 2 illustrates another 
10 embodiment of the invention as circuit 200 wherein the CFA has alternating columns of a 
particular color. In this case, a single shutter control line for each color couples with the 
shutter inputs of all photosites for a given color in multiple columns. 

Although both Figures 1 and 2 show the sensor circuit having three colors (e.g., 
red, green and blue), the sensor circuit may also be configured with other combinations of 
15 two or more colors in its CFA, there being different colors available other than red, green, 
and blue, such as cyan, yellow, and magenta. 

The embodiments of the invention in Figures la and 2 receive a number of shutter 
control signals, an example of which is illustrated in Figure lb as pulse type signals 
having different time division multiplexed pulses. The shutter control signals in Figure 
20 lb control an electronic shutter (not shown) inside each photosite. The electronic shutter 
may be a transistor switch which when turned on will allow the formation of light- 
generated signals inside a capacitor in the photosite. To read out the light-generated 
signal, row or column select signals are supplied to the readout circuitry in the photosite 

Our File No: 042390.P6413 -7- 
Express Mail No: EM560644430US 



that causes the light-generated signal to appear (for example, as an analog voltage) at an 
output port of the photosite. 

In the embodiment of Figure lb, the shutter control signals are pulses which go 
active simultaneously for all colors in the sensor circuit, but which end at different times 
5 depending on the spectral content of the illuminant. For example, the pulse widths for 
the red, blue, and green shutter control signals shown in Figure lb may have a ratio 
relationship related to the color temperature or spectrum of the illuminant. There are 
many techniques available for determining the color temperature or color spectrum of the 
illuminant. 

3 10 One technique for obtaining the color spectrum of the illuminant and the relative 

ft? 

*; pulse timing of the shutter control signals is briefly described below. First, the pulse 

■si 

width for a desired color's shutter control signal is set to be proportional to the ratio of the 
*J measured light energy for a base color to the measured light energy of the desired color 

& during a given exposure time. The sensor array is exposed to an object having uniform 

J 15 reflectivity across all colors (e.g., a 20% gray card) for an initial exposure time that is the 
3 same for all photosites. Next, the measured light energy for each color is obtained by, for 

example, measuring the output of a number of photosites having the same color filter. 

For example, if the illuminant were a tungsten bulb such that the energy content at blue 

was considerably less then the energy content at red, then uniform shutter timing would 
20 result in an undesirably low detected signal level for the blue frequency. The blue signals 

(or blue color channel) would thus exhibit a lowered dynamic range as compared to the 

red signals (red color channel). 
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One color is selected as the base color, such as, for example, the color with the 
highest intensity. Thus, if the pulse width for the blue shutter control signal is set to tblue 
= 1 time unit, then the pulse with t x for a color X is given by t x = tblue * 
Bluemeas/Xmeas where Blue mea s and X me as are proportional to measured light energy 
5 signals for the base color Blue and desired color X, respectively. As shown in Figure lb, 
the two pulses with the smaller widths (e.g., blue and green) are then divided into set of 
sub-pulses, such that the sum of the widths of the sub-pulses equal the determined pulse 
width. For example, red has the longest pulse width tR with a total integration time of 
tRed ^ at ls ec l ua l t0 % ^ e integration time for the image capture. Green has the second 

10 longest pulse width split into four sub-pulses, tQj, tQ2> tQ3 and tQ4 with a total 

integration time of tQ reen . Blue has the shortest pulse width split into four sub-pulses, 
tgi, tg2> *B3 an(i l B4 wit k a total integration time of tj$i ue . In one embodiment, each of 
the two smaller pulse widths is split into the same number of sub-pulses (e.g., 4 in Figure 
lb). In another embodiment, the pulse for each of the two colors may be split into a 

15 different number of sub-pulses widths. For example, tg^ ue may be split into 2 sub-pulses 
instead of 4 sub-pulses. By time multiplexing the pulse widths of the integration signals, 
each color is sampled through-out the total integration time, reducing the effects of 
motion effects introduced by the shorter integration times being sampled only in one 
portion of the time period of the longer integration times. 

20 The measured energy levels for each color are obtained using the same exposure 

time, but because the illuminant may have non-uniform intensity over the different 

colors, the measured energy levels for different colors may be different. For this 

example, since the intensity of the illuminant is highest at blue, the total shutter control 
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signal pulse widths for red and green would be larger than for red, in proportion to the 
ratios described above. This allows the photosites capturing red and green to be 
integrated over a longer period of time. 

The spectral information may be normalized prior to further processing of the 
5 digitized light-generated signals, in that the normalized energy value: 

Y 

\^ _ meas 

for each color is integrated over the pulse width or integration time t x to obtain the total 
number of photons detected in the interval for the particular color. 

Another technique for obtaining the spectral ratios needed for the relative timing 

10 of the different shutter control signals uses a histogram of digital image data received 
from the sensor circuit. The needed information about the spectral content of the 
illuminant may be obtained after exposure of the sensor circuit, uniform in time across 
the different colors. This should yield for a given detected value (digital) the total 
number of photosites of the same color that detected that value. The histogram will thus 

15 present a distribution of photosites versus color energy, for a given scene and illuminant. 
By selecting a statistic, such as 95% of the cumulative distribution of pixels, the digital 
values for the different colors can be used to define a ratio that will be multiplied by a 
predetermined pulse width to give the desired pulse width of the particular color. 

Some of the advantages of the embodiments of the invention may be illustrated by 

20 the following example. An optical subject such as the 20% reflective gray card typically 
reflects 20% of all incident energy substantially uniformly across all colors. A sensor 
circuit in accordance with an embodiment of the invention is placed in front of the 20% 
gray card so as to capture an image of the gray card. If the illuminant were, for example, 
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daylight, such that the incident optical energy at each color was substantially the same, 
then a single exposure with uniform shutter timing (corresponding to equal pulse widths 
for the different color shutter control signals) would result in the same amount of energy 
collected by each photosite. This presents the ideal illuminant situation. 
5 By using independent electronic shutter timing for the blue and red channels to 

increase the exposure time for the blue channel relative to the exposure time for the red 
channel, the analog signal level for the weaker channel may be increased prior to A/D 
conversion thus allowing greater dynamic range in the weaker channel. 

A system embodiment of the invention as an imaging or image capture system is 

10 illustrated in Figure 3. The system includes an optical interface that directs light reflected 
from an optical subject to a sensor circuit 300. The sensor circuit 300 includes a color 
sensor array as in any one of the circuits 100 and 200 described earlier. The sensor 
circuit 300 receives a number of independent shutter control or integration timing signals 
(one per color) from the control unit 308. The control unit 308 may be implemented as a 

15 hard- wired logic circuit, or as a programmed processor with a suitable I/O peripheral, and 
may or may not be located in the same IC containing the sensor circuit 300. 

The control unit 308 receives exposure time values and range measurements from 
block 304 that are used to define the shutter control timings. In block 304, the color 
temperature of the illuminant may be automatically determined as described above, or 

20 alternatively may be set manually by the user. The timing for each color may be 

automatically computed and provided, either by hardwired logic circuits or perhaps from 
an I/O peripheral of a programmed processor, in one or both of the block 304 and control 
unit 308. 
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The embodiment shown in Figure 3 also provides that sensor signals from the 
sensor circuit 300 be transferred to signal processing unit 312. In one embodiment, the 
sensor signals may be in analog form to be converted into digital form by an A/D 
converter in the signal processing unit 312. Alternatively, the A/D conversion units may 
5 be part of sensor circuit 300 so that the sensor signals passed to unit 3 12 are digital. In 
either case, the A/D conversion units may be located on the same IC as the sensor array. 

In addition, the unit 312 may be configured to perform digital image processing 
such as noise suppression and color space conversion. Such digital processing by unit 
312 may be performed by a programmed processor or by dedicated hardwired logic 
10 circuits. 

After image data has been prepared by unit 312, the data may be stored in a data 
storage 316 which may be any conceivable type of storage device suitable for storing 
digital data. Modern examples include a non-volatile random access memory and a 
rotating media device such as magnetic and/or optical disk storage. A data link interface 

15 320 permits the image data to be transferred outside the image capture system to, for 
example, a desktop computer via a serial communications link. 

The embodiments of the invention described above may be used to assist in 
correcting for manufacturing variations that induce a type of imbalance in the different 
color channels. For example, the optical filters for a given color channel may have 

20 unequal bandpass properties due to differences in the CFA between production batches. 
Also, the optical components of the imaging system may present non-uniformity across 
the various colors. Such variations add to the imbalance between the color channels that 
is created by illuminants having non-uniform intensity. The independent shutter control 
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of the various embodiments of the invention helps to equalize the light-generated analog 
signals between the different color channels, and therefore also help reduce the effects of 
the manufacturing variations. 

Another advantageous feature of the circuits 100 and 200 appears when the sensor 

5 array is implemented as a single chip. This allows simultaneous rather than sequential 
duration exposures for all colors, so that the imaging system that incorporates the single 
chip sensor array may yield higher quality color images of a moving scene. 

Figure 4 illustrates one embodiment of control 308 configured to generate 
different integration times with an integration time generation unit 402, a red channel 

10 pulse generation unit 406, a green channel pulse generation unit 410, and a blue channel 
pulse generation unit 414. A red duty cycle register 404, a green duty cycle register 408, 
and a blue duty cycle register 410 are connected to red channel pulse generation unit 406, 
green channel pulse generation unit 410, and blue channel pulse generation unit 414, 
respectively. Each of the duty cycle registers are programmed to a particular duty cycle 

15 to generate the proper pulses for that color. The pulse generation units generate the set of 
sample signals for the sub-pulses that make up the complete pulse. Integration time 
generation unit 402 provides the total integration time for the capture of each frame. 

To summarize, the embodiments of the invention described above present the 
design of an improved and novel color image sensor circuit that features independent 

20 electronic shutter control for each color channel. Of course, the embodiments of the 

invention described above are subject to other variations in structure and implementation. 
For example, semiconductor IC fabrication techniques other than standard CMOS may be 
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used to implement the different embodiments. Thus, the details above should be 
interpreted as illustrative and not in a limiting sense. 

In the foregoing specification, the invention has been described with reference to 
specific exemplary embodiments thereof. It will, however, be evident that various 
modifications and changes may be made thereto without departing from the broader spirit 
and scope of the invention as set forth in the appended claims. The specification and 
drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. 
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CLAIMS : 

What is claimed is: 



1 



1. 



A method comprising: 



2 



determining a first integration time for a first color channel; 



3 



determining a second integration time for a second color channel, where the 



4 second integration time is shorter than the first integration time; 



5 



dividing the second integration time into a set of integration times; 



6 



integrating a first sensor over the first integration time; and, 



7 



integrating a second sensor over the set of integration times. 



1 2. The method of claim 1, where dividing the second integration time into the set of 

2 integration times includes dividing the second integration time by a predetermined 

3 number to generate the set of integration times, where a sum of all integration times in 

4 the set of integration time equals the second integration time. 



2 integration times includes integrating the second sensor over each of the set of integration 

3 times. 

1 4. The method of claim 3, where integrating the second sensor over each of the set 

2 of integration times includes generating a set of sample signals to the second sensor, 

3 where each sample signal in the set of sample signals corresponds to a respective 

4 integration time in the set of integration times. 
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1 



3. 



The method of claim 1, where integrating the second sensor over the set of 



1 5. The method of claim 1 , where the set of integration times are dispersed 

2 throughout the first integration time. 

16. An article comprising a computer readable medium having instructions stored 

2 thereon, which when executed, causes: 

3 determination of a first integration time for a first color channel; 

4 determination of a second integration time for a second color channel, where the 

5 second integration time is shorter than the first integration time; 

6 division of the second integration time into a set of integration times; 

7 integration of a first sensor over the first integration time; and, 

8 integration of a second sensor over the set of integration times. 

1 7. The article of claim 6, where division of the second integration time into the set of 

2 integration times includes division of the second integration time by a predetermined 

3 number to generate the set of integration times, where a sum of all integration times in 

4 the set of integration time equals the second integration time. 

1 8. The article of claim 6, where integration of the second sensor over the set of 

2 integration times includes integration of the second sensor over each of the set of 

3 integration times. 

1 9. The article of claim 8, where integration of the second sensor over each of the set 

2 of integration times includes generation of a set of sample signals to the second sensor, 

3 where each sample signal in the set of sample signals corresponds to a respective 

4 integration time in the set of integration times. 
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1 10. The article of claim 6, where the set of integration times are dispersed throughout 

2 the first integration time. 

1 11. An apparatus comprising: 

2 an image sensor having a first sensor for a first color channel and a second sensor 

3 for a second color channel; and, 

4 an control unit coupled to the image sensor, the control unit having: 

5 a integration time generation unit; 

6 a first color channel pulse generation unit coupled to the first sensor; 

7 a first duty cycle register coupled to the first color channel pulse 

8 generation unit; 

9 a second color channel pulse generation unit coupled to the second sensor; 

10 and, 

11 a second duty cycle register coupled to the second color channel pulse 

12 generation unit; 

13 where the first color channel pulse generation unit generates a set of sample signals to the 

14 first sensor based on a value contained in the first duty cycle register. 

1 12. The apparatus of claim 11, where the second color channel pulse generation unit 

2 generates a second set of sample signals to the second sensor based on a second value 

3 contained in the second duty cycle register. 

1 13. The apparatus of claim 1 1 , where the set of sample signals are a set of pulses 

2 having a duty cycle based on the value contained in the first duty cycle register. 
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1 14. The apparatus of claim 11, further comprising a signal processing unit coupled to 

2 the first image sensor to receive an output. 

1 15. The apparatus of claim 14, where the signal processing unit receives a set of 

2 sampled signals from the first sensor based on the set of sample signals and is configured 

3 to combine the set of sampled signals into a single output. 
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ABSTRACT 



A method including determining a first integration time for a first color channel 
and determining a second integration time for a second color channel, where the second 
integration time is shorter than the first integration time. Then, the second integration 
time is divided into a set of integration times. A first sensor is integrated over the first 
integration time; and, a second sensor is integrated over the set of integration times. An 
apparatus for performing the method. 
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